Automatic Evaluation of Computer Generated Text: Final Report on the TextEval Project

The report describes computational and psychological experiments carried out under grant number GR/H29179 from the Science and Engineering Research Council of Great Britain. These would not have been possible without the gracious assistance of Ian Mason of Heriot Watt University, Edinburgh. We aimed to assess the usefulness of a new technique for the evaluation of translation quality, comparing human rankings with automatic measures. The basis of our approach is the use of a standard set and the adoption of a statistical view of translation quality. This approach has the ability to provide evaluations which avoid dependence on any particular theory of translation, which are therefore potentially more objective than previous techniques.